Back to proposals overview - program

---

Jenkins and automatic multi-branch GIT support

Abstract:

We in HP had a common problem for developers: we had a lot of people working on one product, and CI for that product’s main “trunk” or “master” branch often got dirty (due to bad commits) and it became very problematic for developers to work together. The main reason for this is that even with full developer cooperation and no human-errors, it’s not feasible for them to run ALL TESTS before every push, and there are hundreds of push operations a day.

Our solution to the problem is to provide CIaaS: Continuous Integration as a Service, and we wanted to do this over Jenkins, which is our current CI platform.

With this concept, developers and teams can work on their branches, and get feedback from the complete CI cycle, during development, per push. Once they’re ready to merge with master, they merge master to their own branch, see the results of CI on their branch, and when it’s ok they merge back. Working with this methodology guarantees a green master branch, continuously.

Some challenges: * Jenkins has no concept of branch, which has implications: * Jenkins' standard abstraction level “Job” is not branch aware, and its history shows all branches * Email notifications will be sent to people from previous commits, who are unrelated to your branch: If branch “a” breaks a test and branch “b” doesn’t and “a” is built followed by “b”, “b” will claim that it has fixed “a” and send notification to committers * Similarly anything that reports cross-build (statistics, trends) will show mingled results for all branches. * Jenkins has no concept of pipeline, just jobs that trigger one another, so understanding the execution flow of one branch pipeline is difficult for developers, If we give developers a tool to monitor their branches pipeline, where it’s easy to get confused with other branches, we’re doomed to fail. * There is nowhere to see the current status of a pipeline for a branch, because jobs are shared and can belong to different pipelines at one point in time, so their graphical representation is meaningless.

What is our solution?

Our solution is reminiscent of the Hydra, the mythological snake creature with multiple heads: Upon creating a new branch, the git hook creates a new “head” for the pipeline, with a parameter specifying the branch and repository names. Every push on a branch causes a hook to execute and trigger the relevant head of the pipeline, which passes the branch parameter downstream down the entire pipeline. The pipeline itself stays the same for all hydra “heads”.

It is worth to note that our head job is that which sends the emails and does the initial code pull operation, and it is branch-aware, meaning we email the correct people for the correct changes.

We use a trick to make our pipeline branch aware: matrix jobs and dynamic axis plugin allow us to create matrices of size 1, with inner jobs that are generated dynamically that run for the specific branch currently being built. Thus each inner-job is completely branch aware – it always runs in the context of the same branch.

For jobs that are not context aware we have a build-filter that allows you to select values from a drop-down list of all branches that have been built by this job.

To understand the pipeline we have a build-stream-tree plugin that shows the entire execution flow of anything built, including specifically executions of a pipeline for a specific branch. This view is highly customizable, allowing us to “hide” the matrix wrappers from the trick mentioned above, and other complications."

Speaker:

Nathan Grunzweig , HP

Nathan has been and developer and CD Admin for past 3 years at HP.